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ABSTRACT 


A major performance challenge in developing a massively multi-user virtual 
world is network scalability. This is because the network over which entities 
communicate can quickly develop into a bottleneck. Three critical factors: bandwidth 
usage, packets per second, and network-related CPU usage, should be governed by the 
number of entities a given user is interested in, not the total number of entities in the 
world. The challenge then is to allow a virtual world to scale to any size without an 
appreciable drop in system performance. 

To address these concerns, this thesis describes a novel Area of Interest Manager 
(AOIM) built atop the NPSNET-V virtual environment system. It is a dynamically sized, 
geographical region based, sender-side interest manager that supports dynamic entity 
discovery and peer-to-peer entity communication. The AOIM also makes use of tools 
provided by the NPSNET-V system, such as variable resolution protocols and variable 
data transmission rate. 

Performance tests have shown conclusively that these interest management 
techniques are able to produce dramatic savings in network bandwidth usage in a peer-to- 
peer virtual environment. In one test, this AOIM produced a 92% drop in network traffic, 
with a simultaneous 500% increase in world population. 
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1. INTRODUCTION 


A. THESIS STATEMENT 

It is possible to integrate an area of interest management (AOIM) scheme into a 
peer-to-peer virtual world that supports dynamic entity discovery. Furthermore, by 
offering dynamic class and protocol behavior control mechanisms, this AOIM can offer 
improved scalabihty over a standard interest management scheme. 

B. MOTIVATION 

Anyone that takes the time to study any of the multi-user virtual worlds currently 
in use today, such as Everquest™ by Verant™ Software (www.verant.com), or Team 
Fortress™ by Valve Software™ (www.valvesoftware.com), wiU undoubtedly come away 
impressed. These products whisk users away to a fantasyland where they not only get to 
slay dragons, or defend Earth against an alien hoard, but they can do these things in a 
world populated with other real people. They can work together as a team, hey can 

compete against other users for the same goal, or they can go head to head trying to 

eliminate each other. The possibihties are boundless. 

With their impressive 3D graphics, and 3D spatial sound, it is easy to assume that 
the biggest challenge facing the developers of these worlds is optimizing the graphics and 
sound programming to maintain an acceptable frame rate, as well as smooth flowing 
sound. That assumption is incorrect. The computers of today are more than powerful 
enough to handle real time, near photo-realistic graphics, and spatial audio. In fact, the 

transistor count of a modem 3D graphics chip alone outstrips those of the most powerful 

CPUs of only a few years ago. This trend of faster more powerful computers will not end 
anytime soon. 

The single biggest performance challenge to anyone developing a massively 
parallel multiplayer world is the network programming. The reason for this is that the 
network over which the entities in the world must communicate can quickly develop into 
a bottleneck. Entities communicate with each other by sending informational packets 
across the network. These packets contain everything the other players in the world need 

to know about the entity sending the packet, such as position and appearance. In the 
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unicast case, network traffic is 0(N ), where N is the number of entities in the world. As 
new players join the world, they will begin trying to communicate with aU other entities 
in the world, by sending packets over the network, while at the same time trying to 
receive this same information from everyone else. It will not take long for this bottleneck 
to manifest itself in the form of dropped packets. 

As an example, if a virtual world that is mnning at 30 frames per second has 32 
players in it, each sending a single position-packet per frame, the net result will be 1024 
packets per frame put on the network, which equates to 30,720 packets per second. If the 
number of players in the world increases from 32 to 300, the number of packets per 
second will increase to 2,700,000. Should the number of players climb to 2000 (a 
common number of players found on any of Everquest’s many servers at any given time) 
the packets per second put on the network will increase to 120,000,000. This is just the 
number of packets; the bandwidth used is determined by multiplying packets per second 
by the packet size. Even if the packet size is very small, say for example 50 bytes, the 
amount of data traveling the net wiU stiU quickly overwhelm the fastest network. In the 
Everquest example given above, the 2000 players would attempt to send almost 6 
gigabytes of data per second! Obviously, this is not going to happen, and the fact that 
most users of these online games are stiU using dial-up connections and 56K modems 
only magnifies the problem. 

Three critical network factors greatly help determine the performance of an online 
virtual world. They are bandwidth usage, packets per second, and network related CPU 
usage. In a multi-user virtual world, these factors should be governed by the number of 
entities a given user is actually interested in, not the total number of entities in the world. 
As evidenced in the earher Everquest example, just increasing available bandwidth to 
individual homes via DSE or cable nixiems is not enough. Most users in a virtual world 
will StiU see their systems have unacceptable performance when the number of users in 
the world grows above even a smaU number. The chaUenge then is to aUow a virtual 
world to scale to any size (or at least to several thousand users) without an appreciable 
drop in system performance. This is the motivation for area of interest management. 
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C. DEFINITION 


AOIM is a general term used to describe any filtering process applied to data 
packets to ensure they are used only when and where they are actually required. There 
are many AOIM schemes in existence which generally fall into one of two categories: 
send-side or receive-side. 

Receive-side filters receive every packet on the network and then determine each 
packet’s relevance. If the packet is pertinent to an entity located on that local host it is 
passed along to that entity, otherwise the packet is discarded. An advantage to a receive- 
side interest management scheme is the simphcity with which packets are sent. No a 
priori knowledge of what entities occupy the world or where they are located is required. 
However, this type of interest manager does not address bandwidth conservation, and that 
is its major disadvantage. 

Send-side filters attempt to send packets only when and where they are required. 
An advantage to an interest management scheme of this type is a large reduction in 
bandwidth usage. A disadvantage is the complexity of calculating in real time where to 
send each packet. 

D. APPLICATION 

The main challenge in implementing interest management into a peer-to-peer 
virtual environment is data consistency and AOIM coordination. In a hue peer-to-peer 
system, each client has its own copy of the world at large (interest database), or at least 
that portion of the world with which it is currently concerned. In effect, it is a large 
distributed database that must be kept consistent. If the AOIM of one chent in the world 
decides to make a change, that change must quickly propagate to every other AOIM in 
the world to ensure no loss of data. In a virtual world that allows for dynamic entity 
discovery and loading, this is doubly difficult because new chents and entities can leave 
or join the world at anytime. 

NPSNET-V is a component based, dynamically extensible, scalable framework 
for creating cross-platform, persistent virtual worlds. NPSNET-V eschews the more 
common chent-server architecture in favor of a mostly serverless peer-to-peer approach. 
NPSNET-V fuUy supports dynamic entity discovery, and area of interest management 
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has yet to be implemented in this type of environment. This makes NPSNET-V a perfect 
candidate for performance testing of an area of interest manager supporting dynamic 
behavior control mechanisms. The area of interest management scheme developed in this 
thesis, and integrated into NPSNET-V, is primarily a send-side system. 

E. APPROACH 

This thesis shows that the apphcation of an area of interest manager greatly 
reduces the amount of network bandwidth used by a peer-to-peer virtual world supporting 
dynamic entity discovery, which in turn translates directly into much more scalabihty. 

There are three main ways this AOIM accomphshes its goals. The first is through 
extensive use of dead reckoning, which is the process of calculating an entity’s current 
position based on its last known position, speed, and heading, as well as elapsed time 
since the last known position. The second is by utilizing variable resolution data packets 
that allow the area of interest manager to make level rf detail adjustments of data packets 
on the fly. The third is by using multicast and dynamically sized geographic zones, each 
assigned one or more multicast addresses, allowing an efficient method of sender side 
packet filtering. 

F. THESIS GOALS 

• Develop a complete, dynamic, multi-function, area of interest management system 
that uti liz es dead reckoning, variable packet resolution, and dynamically sized 
geographic multicast regions. 

• Implement this AOIM scheme into NPSNET-V, such that it is capable of managing 
dynamic entities and dynamic entity types. 

• Gather benchmark data to show this AOIM scheme allows a much higher level of 
scalabihty than an AOIM without support for dynamic loading. 

• Provide future students with a basehne set of Java classes defining an AOIM that can 
be adapted to schemes other than geographic zones. 
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G. THESIS ORGANIZATION 


The organization of chapters in this thesis is as follows: 

• Chapter I: Introduction. Contains the thesis statement, an overview of the problem 
and some possible solutions, a few important definitions, and fists the goals this thesis 
strives to obtain. 

• Chapter II: Background. Review of prior and current work in this area. Describes 
other area of interest management systems, fisting some of their strengths and 
weaknesses. Also discusses NPSNET-V, describing in detail its unique features as 
they pertain to area of interest management. Finally this chapter discusses IP 
multicast as it applies to interest management. 

• Chapter HI: Approach and architecture. Discusses the approach this thesis takes in 
the development of a send-side area of interest management scheme, as well as the 
hurdles that were encountered and overcome. Also discusses in detail all algorithms 
used for the area of interest manager, whether borrowed from another source, or 
developed specifically for this thesis. 

• Chapter IV: Experimentation and analysis. Compares NPSNET-V performance, 
scalability, and bandwidth usage both with and without the area of interest 
management scheme developed in this thesis. Discusses in depth each specific test 
and the network conditions under which they were mn. 

• Chapter V: Conclusions and future work. Discusses the strengths and weaknesses of 
the interest management scheme developed by this thesis. Discusses specific changes 
to the system that could improve performance. Easily, this chapter discusses several 
thesis level modifications that could be made to this system. 
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n. RELATED WORK AND BACKGROUND 


A. INTRODUCTION 

This chapter provides background information for several previous area of interest 
management systems. It also discusses NPSNET-V, including many important 
definitions used throughout the rest of this thesis. Lastly, this chapter discusses IP 
multicast as it applies to AOIM systems in general. 

B. PRIOR WORK 

Many virtual environment systems were developed before the inception of NPSNET- 
V. These can be roughly broken-up into two distinct groups: chent-server and peer-to- 
peer. 

I. Client-server systems 

As the name imphes, the following systems primarily use chent-server 
architectures. NPSNET-V is mostly peer-to-peer, and the client-server systems discussed 
here are included for completeness. 

a. BrickNet 

The Bricknet too lk it was originahy designed to facihtate the rapid 
development of networked virtual worlds. The worlds would operate on networked 
workstations and form a loosely coupled system. The designers envisioned that Bricknet 
would be used to constmct multi-user games, concurrent engineering systems, and other 
asynchronous GUI environments. 

The entity communication process in BrickNet is as foUows: Chents pass 
packets via UDP to their server. The server then decides where those packets need to go. 
If other chents on that same server need the packets, the server will transmit them directly 
there. If chents that are being serviced by other servers need the packets, then the server 
will pass the packets via UDP to the appropriate server and they in turn wih pass them on 
to their respective chents. 

BrickNet implements interest management by having each chent register 
with the server when it connects to the net. As part of this registration process, the chent 
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passes a list of objects the given entity is interested in to the server (Singh 1994). The 
server will then pass the appropriate packets along as necessary. For an example of how 
this may work, consider a player joining the world in an aircraft. The player’s chent wiU 
register with the appropriate server and may express an interest in other aircraft only. As 
the player flies around the world and reports to its server any actions it takes, such as 
turning, climbing, or firing, the server wiU pass the information on. It is important to 
note, however, that the information is only passed to other entities that have expressed an 
interest in aircraft. If the player were to fly over a large field containing several soldiers, 
he would not receive any packets from them and would consequently not know that they 
were there. However, if the soldiers had expressed an interest in aircraft, they would 
receive the aircraft’s packets, and would therefore be able to detect it. Again, this system 
is completely chent-server. 

The advantage to this system is simphcity. The chent-server paradigm is 
weh understood. The biggest disadvantage is scalabihty. To scale to the size that 
NPSNET-V aspires to requires something other than a chent-server architecture. 

b. RING 

RING is another chent-server system that provides for multi-user virtual 
worlds. It was designed to support densely occluded virtual environments, such as 
buildings or cities, with a large number of users. RING is a good choice for building a 
walkthrough, but would not lend itself weh to a virtual world with wide-open spaces, 
such as a flight simulator. 

RING uses UDP datagrams exclusively, avoiding Multicast altogether. 
The area of interest management approach taken by RING is that as entities change state 
they send update packets to their respective server. The server in turn performs hne of 
sight calculations, and then passes the packets along to only the entities that it has 
determined can actuahy see the recently changed entities. This system works best in a 
“large, densely occluded environment." (Funkhouser 1995) In other words, if everything 
is in the hne of sight of everything else, then there wih not be much packet cuhing taking 
place. 
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The biggest drawback to this system is its reliance on a server hierarchy 
and the associated problem of reliabihty/survivabihty. If one server crashes, the whole 
system crashes. Another, albeit lesser problem with the server hierarchy is that a data 
packet may have to pass from the sending entity through multiple servers, each one 
performing their own calculations, before finally being transmitted to its destination, thus 
adding latency. 

c. SPLINE 

SPLINE provides an architecture for creating multi-user interactive 
environments based around a shared world model. It was intended for building large- 
scale social environments with many simultaneous users. 

SPLINE is a region-based system that makes extensive use of servers. In 
SPLINE the world is divided into non-uniform areas called locales. “The concept of 
locales is based on the idea that while a virtual world may be very large, most of what 
can be observed by a single user at a given moment is nevertheless local in nature” 
(Barms 1996). There are three key characteristics shared by all locales: separate 
communication channels, local coordinate systems, and arbitrary geometry and 
relationships. 

SPLINE, like NPSNET-IV, implements communication by assigning 
individual multicast addresses to locales. Unlike NPSNET-IV, however, locales can vary 
in size and shape. This allows locales to exactly fit the world at large with no wasted 
space. It also allows for a very efficient use of multicast addresses (a scarce resource). If 
a virtual world is not uniformly populated, then all of the unpopulated portion of the 
world can be represented as one locale. An example of this is an aquarium simulation 
where most of the fish are schoohng in one part of the tank. In this respect, SPLINE is 
one of the most efficient systems reviewed here. 

The single apparent weak spot with SPLINE is the fact that it relies 
heavily on communication servers. In fact, every time a new locale is created, a new 
locale server is created right along with it. Any SPLINE process that becomes interested 
in a locale will query its server to receive the most up-to-date information about the 
objects currently in that locale. It is important to note, however, that entity-to-entity 
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communication is via multicast. The servers are only used to allow for lateeomers and 
world eonsisteney. As in any system that reties heavily on servers though, sealabitity is 
likely to be of eoneem, as well as reliability sinee servers are single points of failure. 

2. Peer-to-peer systems 

The following systems are primarily peer-to-peer, but may make use of bootstrap 
servers. NPSNET-V falls into this eategory. 

a. DIVE 

Like SPLINE, DIVE was designed as a fully distributed system to enable 
large numbers of users to interaet in a shared environment. A further design goal was 
that eaeh user also be able to interaet with the environment itself. 

“DIVE foeuses on peer-to-peer multieast eommunieation as opposed to a 
‘elassie’ etient-server model” (Ereeon 1998). However, the system does make use of an 
important server eatied the “Co lli sion Manager” in its implementation of area of interest 
management. 

DIVE makes use of two eoneepts eatied foeus and nimbus. Eoeus is 
defined as a geographie area around an entity within whieh that entity is able to deteet 
other objeets in the world. If something enters an entity’s foeus, that entity will be able 
to see and hear it. A nimbus is also a geographie area around an entity. Objeets in an 
entity’s nimbus are able to see and hear the entity. This interesting approaeh allows for 
situations where an airplane is inside a radar site’s nimbus, while at the same time; the 
radar is outside of the airplane’s n im bus. This allows the airplane to deteet the radar, 
without the radar deteeting it. This is very mueh how things work in the real world. 

There is, however, a drawbaek to this system, and it is a big one: 
sealabitity. The co lli sion manager eonstantiy ealeulates the foeus and nimbus co lli sions 
for every entity in the world. Sinee these ealeulations are 0(N ), they are elearly the 
limiting faetor in the sealabitity of any DIVE apptieation. More importantly, this server 
is a single point of failure; without it, the system eannot fiinetion. 

Retianee on the co lli sion manager not-withstanding, entity-to-entity 
eommunieation in DIVE is peer-to-peer. When an entity needs to make a request, or 
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realizes that it is missing a needed data packet, it sends a request over the appropriate 
multicast channel. Utihzing a proprietary algorithm to estimate packet round-trip-time 
and calculate an appropriate timeout, the entity that is closest to the requesting entity will 
respond with the needed information. This response is sent over multicast as well, thus 
ensuring that other entities considering responding to the request, will reahze that another 
entity already has, and will not themselves respond. This prevents an explosion of 
responses for each single request that is sent (Frecon 1998). 

b. NPSNET-IV 

NPSNET-IV, the precursor to NPSNET-V, was designed primarily to 
support virtual battlefield simulations over large expanses of terrain. As such, there is no 
concept of height or depth. A virtual world built on NPSNET-IV will essentially be flat. 
This limitation would lik ely preclude using NPSNET-IV for the development of a flight 
simulation or any similar system. 

NPSNET-IV makes extensive use of multicast in implementing area of 
interest management (Macedonia 1995). In this system, the world is broken up into a 
grid of hexagonal cells. Each of these cells is associated with a multicast address. When 
an entity first enters a new cell, it subscribes to the appropriate multicast address. Next, 
the entity will send a join PDU to the cell, and the “oldest” entity in the cell will respond 
with state information for every entity currendy in the cell. This response is made over a 
TCP/IP connection to ensure rehabihty. When an entity moves into yet another cell it 
will send a leave PDU to the old cell, and the process repeats again. 

The biggest drawback to this system is that it is fairly apphcation specific. 
The reason for this is that the hexagonal cells used to subdivide the world are not 
dynamically resizable. Their size is determined at compile time. Much experimentation 
went into the determination of the optimal size to make the cells for the initial NPSNET- 
IV demonstration, and any reuse of NPSNET-IV for a different apphcation would require 
similar experimentatioa If every entity in the world were to move into one ceU, this 
system could not adapt. 
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c. Three-Tiered Interest Management Scheme 

The ‘Three-Tiered Interest Management Seheme’ is another AOIM 
developed at the Naval Postgraduate School (Abrams 1999). This system uses 
dynamically sized zones, represented by multicast addresses, as the first tier. The second 
tier uses data received from the first tier to, “create a protocol independent perfect match 
between a chent’s interests and the environment.” The third tier takes the data passed 
through the first two tiers and filters it based on specific protocols. The single largest 
drawback to this system is that the second tier was implemented by assigning a separate 
multicast address to each entity in the world. Since multicast groups are a scarce 
resource, this clearly causes some serious scalabihty problems. 

C. NPSNET-V BACKGROUND 

The NPSNET research project was begun in 1990 at the Naval Postgraduate 
School. The project is a continuing test bed for advanced apphed research in networked 
virtual environments (McGregor 2001). NPSNET-V is the fifth and latest incarnation of 
this research. The goal of NPSNET-V is to be “a framework for fuUy distributed, 
component based, persistent, networked virtual worlds, extensible at mntime and scalable 
to i nfi nite size on the Internet.” NPSNET-V is a large and complex architecture and 
describing it in its entirety is far beyond the scope of this chapter. The interested reader 
is encouraged to point their web browser to http://www.npsnet. 0 rg/~npsnet/v/index.htn 1 l 
or to read the paper: NPSNET-V. A New Beginning for Dynamically Extensible Virtual 
Environments (Capps 2000), for complete information on the NPSNET-V architecture. 
There are, however, three specific components of NPSNET-V, the understanding of 
which, are critical to the AOIM developed in this thesis. They are entities, protocols, and 
entity dispatcher. 

1. Entities 

Entities in NPSNET-V are defined as any “active” component in the virtual 
world. These could include obvious things such as vehicles or avatars, but less intuitive 
things such as terrain elements or physical forces can also be entities. There are three 
types of entities possible in NPSNET-V: masters, ghosts, and shared. 
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Entity masters, as the name would imply, are the actual representation of an 
object in the world. There will only ever be one entity master per object in the world, and 
it will reside on the machine of the user controUing it. They contain the most accurate, 
up to date information concerning the object they represent. 

Entity ghosts represent their master. They approximate their respective master as 
closely as possible. If a user creates a new entity master in the virtual world, every other 
user in that world that sees “it”, will actually be seeing an entity ghost. 

“Shared entities approximate ideal shared state” (McGregor 2001). Good 
examples of shared entities are terrain, or physical models. 

2. Protocols 

When an entity changes state, that change needs to be made known to the virtual 
world at large. Protocols are the component responsible for this. They communicate all 
state changes over the network, via the entity dispatcher. “Current protocols are 

lightweight and composable; each transmits a single element of state” (McGregor 2001). 
Consequently, there will lik ely be multiple protocols per entity. Some examples of 
protocols in NPSNET-V are TransformProtocol and AnimationProtocol. 

3. EntityDispatcher 

This is the NPSNET-V object that deals with network communication. There is 
only one of these per machine in the virtual world, and protocols make use of methods in 
this class to send data packets over the network. Eigure 1, in the following chapter, 
shows the relationship between entities, protocols, and entity dispatcher. 

D. MULTICAST 

Multicast, which is based on UDP, is a form of network communication that 

sends out data non-redundantly across a network graph. Eike UDP, multicast is 

unreliable; u nlik e UDP, however, multicast packets do not go to a specific Internet 

address, but are sent to a group address instead. The process is similar to opening a 
broadcast network connection, with the additional requirement that the user must also 
subscribe to the specific multicast addresses they are interested in. Any packet sent to a 
multicast address will result in that packet being sent to every other computer that is 
currently subscribed. Data reception works exactly the same way. Subscribing to a 
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multicast address wiU result in the subscribed computer receiving every data packet sent 
to that particular multicast address. There is no concept of “send only”, or “receive only” 
where multicast is concerned. If a computer subscribes to a multicast address it will 
immediately begin receiving every data packet sent to that address (even its own). It is a 
simultaneous one-to-many and many-to-one relationship. 

1. Multicast benefits 

There are several benefits to using multicast. The first of these is that there is no 
need for a sender to leep track of subscribers. A user just transmits data packets once, 
and anyone currently subscribed, including the transmitter, receives them; subject of 
course to the fact that multicast is unreliable. This is a powerful feature, but the single 
largest benefit to using multicast is that data filtering is pushed onto the network 
hardware layer and away from the application layer. As network hardware improves, 
becoming faster and more sophisticated, the user will see the benefits immediately and 
automatically. 

2. Multicast drawbacks 

There are also some drawbacks to multicast. There are 268,435,456 multicast 
addresses available on today’s Internet. They range from 224.0.0.0 through 
239.255.255.255. At first glance, this may not seem tike a problem, but as more and 
more virtual worlds pop up, each with possibly thousands of users, using hundreds or 
thousands of multicast addresses, the very real possibility of collisions presents itself. As 
an example, consider two massively multi-player worlds trying to use the same block of 
multicast addresses. Game-two’s data packets arriving at machines mnning game-one 
and vice versa will cause problems. At the very least bandwidth will be wasted, and in 
the worst case one or both games could crash or have unexpected behavior. 

Another problem with multicast is that it is not widely supported on the Internet. 
Many routers have their multicast feature disabled, or only support a lim ited number of 
addresses. In addition, PC network cards currently support only a few simultaneous 
multicast groups in hardware (Abrams 1999). More than this requires pushing some of 
the processing back up to the operating system layer, thereby partially negating the single 
biggest advantage to using multicast in the first place. 
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E. CONCLUSION 


This chapter has outlined several virtual world implementations both peer-to-peer 
and chent-server, along with their assoeiated interest management schemes. It has 
provided the necessary background on NPSNET-V, the architecture that this AOIM is 
integrated with, to understand the rest of this thesis. Lastly, multicast was described 
along with it benefits and lim itations. 
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III. DESIGN AND IMPLEMENTATION OE THE AOIM 


A. INTRODUCTION 

This chapter presents the underlying theory, design, and implementation of the 
geographical area of interest manager. It discusses in detail the requirements that an 
AOIM must meet or exceed to be successful, the design choices made to meet those 
requirements, and lastly the actual implementation. In short, this chapter is the why, 
what, and how of a successful geographical AOIM. 
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Figure I. NPSNET-V packet distribution process 


The following is an overview of the NPSNET-V packet distribution 
process shown in Figure 1. Whenever an entity in the world changes, it notifies one or 
more of its protocols. The protocol in turn queries the entity for the needed data 
regarding its state change. The protocol then asks the AOIM for a distribution set of 
multicast groups where the update data needs to be sent. Next, the AOIM ensures the 
entity has the correct data fidehty and data resolution settings. Finally, the AOIM 
calculates where the data needs to go, builds the appropriate distribution set, and returns 
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it to the protocol. The protocol then passes the distribution set, along with the data packet 
being sent, to the entity dispatcher, which in turn sends it on its way. 


Figure 2 shows a high level breakdown of the AOIM functionality provided by 
the system developed in this thesis. Figure 12, later in the chapter, further expands this 
diagram, and provides more details of the AOIM process. 
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Figure 2. AOIM functionality overview 


B. REQUIREMENTS 

To be successful an AOIM must consume resources proportional to the number of 
entities with which it is actually interested. Bandwidth usage, packets-per-second, and 
network-related CPU usage should be independent of the total number of entities in the 
world (Abrams 1999). 

C. DESIGN OVERVIEW 

The AOIM designed in this thesis is a dynamic, object oriented, geographical 
region based, sender-side, peer-to-peer (mostly), interest manager that supports dynamic 
entity discovery. Every geographical region is represented by one or more multicast 
addresses. The AOIM also makes use of external tools specific to the NPSNET-V 
system, such as variable resolution protocols, and variable data transmission rate. 

I. Dynamic geographical regions 

There are many options available for subdividing a virtual world, ranging from 
simply dividing it in half, to more complex polygonal tessellations. The scheme 
implemented in this work utilizes octrees. This allows the system to collapse or expand 
any or aU regions in the world, on the fly, in an extremely fast and efficient manner. 

An octree, as the name imphes, is a tree type data-stmcture, with one simple mle: 
every node in the tree has either zero or eight children. A node with no children is called 
a leaf node. An octree lends itself very well to the representation of three-dimensional 
space. Initially, the octree will consist of a single leaf node representing the entire world. 
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Figure 3. Octree leaf node 


When the AOIM determines that the world has become too crowded, it can tell 
the octree to subdivide. The leaf node then breaks into eight new octrees, each of which 
is now a leaf node in the tree. 



Figure 4. Octree after one subdivision 

If the AOIM determines one of the new zones has become too crowded, it will 
simply teU that leaf node to subdivide as well. The resulting octree after two subdivides 
might look like Figure 5. 



Figure 5. Octree after two subdivisions 

In the event a subdivided node becomes less crowded after a time, the AOIM can 
just as easily collapse a node. This process lends itself very well to recursion, since each 
new leaf is a duphcate of its parent in form and function. Figures 3, 4, and 5 show the 
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geometric representation of the octree data. Figure 6 shows the underlying tree stmctures 
that the octree class uses for each of the geometric ones. 
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Figure 6. Graph representation of octrees from Figures 3, 4, and 5 


Another benefit to using an octree is the speed with which searches are 
performed. Searching an octree is analogous to performing three binary tree searches for 
each level of the octree that needs searched, one for each of the three axes X, Y, and Z. 
For example, if an entity needs to send an update packet and queries the AOIM for where 
to send the information, the AOIM will perform an octree search based on the entity’s 
position in the world. If the current state of the world is represented by the octree in 
Figure 7, and the entity is in the upper rightmost zone, then the search will need to go 
three levels deep to find the multicast address the entity should transmit on. The first 
pass of the search will determine that the root node is sub-divided. The second pass will 
determine that the appropriate child node has also been sub-divided, and the third pass 
will return the needed information from the correct leaf node three levels deep. 
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Figure 7. Zone containing transnitting entity 


The way the search is conducted is as follows: The AOIM wiU caU a method 
(Octree . whichZone (x,y, z)) in the root octree node, passing along the entity’s 
position (x y z coordinates) in the world. The first thing this method does is check f the 
root node is a leaf; if so the search is complete. If the root node is not a leaf, it 
recursively calls an internal method (Octree . getZone (x, y, z )) that performs a 
search that returns the child node containing the entity. Then that child node has its own 
Octree.whichZone(x,y,z) method called. This process continues recursively 
until the correct leaf node is reached. 

This search is very fast. If there are four levels of subdivision in the world, there 
could be up to 8"^ (4096) separate zones in existence. This search algorithm, however, 
would only need four queries to determine the correct leaf node. In the general case, the 
number of possible zones in an octree is ^ where N is the number of levels in the tree, 
and only N queries are necessary to drill down to the correct node. 

The best case search performance is when the tree is full. The performance of the 
search algorithm in this case is 0(log(H)), where H is the height of the tree. The worst- 
case performance is when the tree only has one non-leaf node per level. In other words, 
at each level in the tree, there is only one node with children (see figure 6). In this case 
performance is 0(H), where again, H is the height of the tree. 

The octree implementation used in this AOIM only stores two pieces of 

information at each node: a String containing the name of the node, and a Boolean 

variable representing whether or not that particular node is a leaf. The node naming 

convention the octree follows is simple and has several advantages. The root node is 

always named “0”. When the root node subdivides, each of its children will take that 
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name and append a single digit to it representing the child node’s index in the array of 
children. For example after a single sub-division the root node will stiU be named “0”, 
and it will have eight children named “00”, “01”, and so on through ”07”. If node “07” 
were to subdivide, then its name would stay the same and it would get eight children 
named “070”, “071”, through ”077”. There are many advantages to this scheme 
including automated name generation, and guaranteed unique names for each zone. In 
fact, just knowing the name of a zone tells you immediately where in the octree the zone 
is, which greatly aids in the debugging effort. 



The AOIM makes use of hashtables to store all multicast addresses and ports over 
which it may need to transmit. The key used by the hashtables is the name of the octree 
zone. The decision to store the multicast addresses and ports separately from each other 
and the octree was made to allow for various degrees of resolution. It is possible to have 
many different hashtables, all containing multicast addresses keyed on octree zone 
names. This allows the AOIM to use one octree and its associated naming scheme to 
send packets when and where it sees fit. One hashtable can be used for the highest level 
of resolution so that packets are only sent to the zone the entity actually resides in, and 
another can be used in a broadcast mode so that packets are sent everywhere. The octree 
remains unaware of these intricacies, only concerning itself with which zone any given 
entity resides. 
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2. Send-side filter paradigm 

The decision to go with a send-side filtering scheme was driven by the design 
goal of NPSNET-V to be useable over a 56K modem connection, and the need for 
bandwidth conservation that goes along with that requirement. The AOIM attempts to 
send an entity’s data packets only where those packets are actually needed; so only 
minimal receive-side filtering is necessary. To do this not only requires the AOIM to 
calculate where in the world every entity is when it is transmitting, but requires that it do 
it in real time as well. This requirement reinforces the choice of the octree/hashtable 
combination described in the previous section. Both data stmctures offer extremely fast 
searches even when they grow very large, allowing for virtually unlimited scalabUity, and 
both data stmctures grow and shrink quickly and easily, allowing for fast, on-the-fly, 
changes to the world. 

3. AOIM with minimal centralization 

The NPSNET architecture is mostly peer-to-peer and this causes some problems 
for a send-side AOIM. A few of the more difficult challenges are: maintaining world 
consistency among aU the players in any given world, keeping track of players joining 
and leaving the world at mntime, and the infamous latecomer problem. 

The latecomer problem occurs in peer-to-peer environments because of the lack 
of a central server with a well-known address. In hue peer-to-peer systems, the state of 
the world is distributed across the machines currently in that world. As users leave the 
world, any world-state information they may have been maintaining shifts to other 
machines on the network, thus maintaining world consistency. The problem occurs when 
new users try to connect to the world. They need the entire state of the world, but there is 
no one place to get it, and just sending a general request for the needed information over 
the network can easily slow it to a crawl as aU the current occupants of the virtual world 
respond to the request, flooding the network with redundant information. The more 
populated a virtual world becomes, or the more user turnover there is, the worse the 
problem becomes. 

The solution employed in this thesis to meet these challenges was to build a 
minimal AOIM server. The AOIM server resides at a well-known network address, thus 
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solving the latecomer problem. It maintains the master copy of the current world 
configuration (octree and hashtables), thus solving the problem of maintaining 
consistency among the players. The AOIM server also registers players as they enter the 
world and de-registers them as they exit. 

The registration process consists of simply storing the address of new users when 
they join the world. These addresses are stored in a Vector, which allows for both 
scalabihty and ease of use. When the AOIM server needs to pass something to the 
individual AOIMs, such as a command to expand or collapse a zone, it simply iterates 
through the Vector to do it. De-registration is simply the removal of the appropriate 
address from the Vector. 

While these capabilities sound very much like chent-server, and not at all lik e 
peer-to-peer, use of the client-server portion of the AOIM is kept to a minimum. The 
AOIM only utihzes the client-server portion of the system in three situations: when first 
connecting to the world, when departing the world, or when there is a configuration 
change to the world (these changes are infrequent). AH data flow among entities remains 
exclusively peer-to-peer. 

4. Implementation details 

Each user in the world will have one AOIM mnning on their machine as part of 
NPSNET-V. When a user runs an NPSNET-V enabled program, one of the first things it 
does is to read an XML configuration file. This file contains, among other things, the 
type of AOIM to use, as well as the IP address of the corresponding AOIM server. The 
AOIM then estabhshes a TCP/IP connection with the server and registers itself. The 
AOIM server responds by logging the IP address of the new joiner, and sending back the 
current octree and all associated hashtables. The AOIM then spawns a new thread that 
waits for the AOIM server to connect to it via a second TCP/IP connection. This gives 
both the server and the AOIM the abihty to initiate communications. 

In a typical client-server setup the chent will initiate communication, the server 
win hear the request, respond to the request, and then enter a wait loop waiting for the 
next chent request. The problem with this is that the server cannot initiate 
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communication on its own. Having two different connections allows the server to push 
information to the chent. This abihty to push is critical to the AOIM’s design. 



Figure 9. Client server architecture 


Once everything just described has taken place no further interaction between the 
AOIM and the server will occur until the configuration of the world changes, or the 
AOIM program terminates. Again, all data communication among entities is tmly peer- 
to-peer. In fact, if the AOIM server crashes, the impact is min im al on the world at large. 
The configuration of the world (the number and arrangement of zones, to be specific) is 
frozen in its current state, and no new players are able to join the world until the server is 
brought back onhne. However, aU the entities already in the world will continue to 
communicate as before, and there will be no indication of a problem to any players 
already in the world. 

When one or more AOIMs decide a zone needs to expand or collapse, they send a 
request to the server. The server responds by locking out aU other requests for the same 
zone change (additional requests are simply discarded). If the request is for a zone 
expansion, the server checks to make sure it has eight new multicast addresses available. 
If it does, it expands the appropriate zone, loads the new multicast addresses and ports 
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into the appropriate hashtable, and then pushes a message to every registered AOIM 
telling them to expand the zone. The AOIMs respond by requesting the new hashtables 
and expanding their respective octrees. In a collapse request the server pushes a message, 
and the clients collapse the appropriate zone. No further action is necessary for a 
collapse. The hashtables maintained by each AOIM wiU stiU contain multicast addresses 
keyed on zone names that no longer exist, but this causes no problems. The AOIM’s 
hashtables are only updated after an expansion. The trade off here is that by continuing 
to store some unneeded information, the AOIM can avoid downloading a copy of the 
hashtables every time a zone collapses. Again, this is aU driven by the desire to conserve 
bandwidth. 

5. Tools provided by NPSNET-V 

The two external tools available to the AOIM to help conserve bandwidth are 
variable entity data transmission rate and variable resolution protocols. 

Variable entity data transmission rate or, data fidelity as it is sometimes called, is 
simple to understand and challenging to implement. The idea is to send exactly the 
number of data packets needed to allow every entity ghost to closely or exactly represent 
their respective entity master. The way this works is as follows: 

All NPSNET-V Entities implement the NetworkTunable interface which 
provides two methods: increaseFidelity () and decreaseFidelity () . 
These methods work just like a volume knob on a radio. Every time 
increaseFidelity 0 is called, the amount of time that passes between each 
heartbeat will decrease, until eventually the entity will send packets at frame rate. This is 
clearly the highest fidelity possible. Conversely, each time decreaseFidelity ()is 
called, it will increase the time between heartbeat packets and the ghost will have to rely 
more and more on dead reckoning. This is obviously a very powerful tool the AOIM can 
use in the fight to conserve bandwidth, and the key to implementing it is dead reckoning. 

The idea of dead reckoning is as follows: given an entity’s last known position, 
course, speed, and the elapsed time since the last known position was reported, the 
entity’s current position can be estimated fairly accurately. Eor example, if an entity at 
position (0, 0, 0) heading straight up the Y axis at a speed of 1 unit per second sends out a 
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data packet, and never changes course or speed again, it will never have to send another 
packet (Figure 10). This is because every time a ghost entity needs to update its position, 
and there are no new data packets available, it can just calculate where it should be. 


Master Entity 


Ghost Entity 


Y=3 


When a ghost entity needs to update its 
position and there have been no new data 
packets since the last update, it calculates the 
following: new pos = old pos + (rate * time) 


T = 2 


Y=2 




After 2 seconds the 
master entity will have 
a position of (0,2) 




At t = 2 the ghost is an 
exact replica of its 
master, even though the 
master has only sent 
one data packet 


T=1 


Y=1 





Ghost needs to update its 
position, but no new data 
packets have arrived. 
Position is calculated using 
dead reckoning. 


Y=0 


T = 0 


Master starts moving 
along the Y axis at a 
speed of 1 unit per 
second and sends 1 
movement packet 




Ghost receives 
movement packet 
and stores the 
velocity and 
acceleration info 


X=0 X=0 

Figure 10. Example of dead reckoning 


A problem arises however if an entity master changes speed or direction, sends a 
new position packet and one or more ghost entities do not receive that packet. In this 
case, the ghost will not be an accurate representation of its master. NPSNET-V employs 
a two-part solution to this problem (Figure 11). The first part is to have each master send 
a periodic position packet (heartbeat) whether it has changed course and speed or not. 
The second part is on the ghost side. When a ghost receives a position packet from its 
master, and the ghost’s position does not match that of its master, it will employ a 
smoothing algorithm to gradually move to the correct position instead of just jumping 
directly there. This is more for aesthetics, or the user experience, than anything else. The 
mle NPSNET-V ghosts employ is to move from their current position to the correct one 
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in 60 frames (approximately 2 seconds). In this way, a person watching a ghost wiU not 
see a sudden, discontinuous jump. 
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and adjusts its speed to Vi 
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The second external tool available to the AOIM is variable resolution protocols. 
NPSNET-V provides a component class called EntityDispatcher. This class has a 
method called setResolution () . This method accepts a Byte as its only parameter. 
This method functions a tittle differently than the methods discussed in the preceding 
paragraphs. It is analogous to a gearshift in a car. Passing 1, as a parameter to the 
method, shifts all protocols to a low resolution, and passing a 2 wiU shift them into high- 
resolution mode. In effect, this method simply changes the precision of the data packets 
that are being sent. In high resolution mode, all the information transmitted in the data 
packets will be double precision. In low resolution mode the information will change to 
single-precision floating point from double-precision, or from a long, to an integer 
representation as appropriate. The trade off here is bandwidth for accuracy. In low- 

fidelity mode, data packets will be much smaller, but ghosts will be less accurate 
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approximations of their masters. Any entities that attempt to interact with these ghosts 
wiU always be a little off. 

6. Dynamic entity discovery 

One of NPSNET-V’s key features is that it supports dynamic entity discovery. In 
other words, a user can create a new entity from scratch, and enter it into a virtual world 
populated with users that have no pre-existing knowledge of this new entity. The new 
entity will look and behave exactly as it should on everyone’s machine, even though it 
has never before existed. 


What enables NPSNET-V and the AOIM to support dynamic entity discovery is 
the Java programming language and a capabihty it provides called reflection. This 
feature allows a program to query an Object, at mn time, about its name, available 
methods, and implemented interfaces. The AOIM makes extensive use of this capabihty. 
When AOIM changes the data fidehty setting, it uses the instanceof operator 
(Holzner 2000) to check if each entity supports dynamic data fidehty. If the entity 
supports it then the appropriate change is made, if it does not, then nothing happens, and 
no exception is thrown. Without reflection the AOIM would not have been able to fuhy 
support dynamic entity discovery. 

D. SUMMARY 


The area of interest management fiinctionahty imbedded in this system is 
summarized in Eigure 12. 
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Figure 12. AOIM process and functionality 
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The AOIM described in this thesis can adapt to a constantly changing 
environment, at mntkne, to maintain network bandwidth usage within acceptable 
parameters. It performs this task in real time with no adverse effect on system 
performance as a whole. The AOIM does this by employing highly efficient data 
stmctures and fast search algorithms that allow for rapid size changes and even faster 
queries. The AOIM allows NPSNET-V to make extensive use of the peer-to-peer 
paradigm, and while there is a chent-server component to the system, its use is kept to an 
absolute minimum. 

The AOIM also makes extensive use of external tools provided as part of the 
NPSNET-V architecture: variable resolution data packets, and variable data transmission 
rate (fidehty). 

Every tool available to the AOIM is useable at mntkne to change how the system 
uses network bandwidth. The changes these tools make occur seamlessly, with the users 
in the world bhssfuUy unaware they have even occurred. 
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IV. PERFORMANCE ANALYSIS 


A. INTRODUCTION 

This chapter describes the performance and rehability testing to which the AOIM 
was subjected. It describes in detail, the test environment and parameters used. Finally, 
the results are analyzed and specific recommendations are made, that suggest how the 
AOIM should respond at mntime to changing network conditions. 

B. TEST OVERVIEW 

There are three distinct groups of tests performed in this chapter. The first group 
is designed to isolate and test the performance of the individual components of the 
AOIM. Performing these tests will allow each component to show their individual 
strengths, working alone, and in conjunction with the other components of the AOIM. 
This will be very helpful in designing the control algorithm that will drive the AOIM at 
mn time (see future work section in Chapter V). It is expected that each component, 
working alone, will cause a significant reduction in total bandwidth usage. It is further 
expected that using the components in concert will show additional reductions in 
bandwidth usage over the individual results. 

The second group of tests is designed to probe the full range of performance of 
the data fidehty component of the AOIM in a fixed world configuration. The purpose of 
these tests is to determine if bytes per second (BPS) decreases in a linear fashion as data 
fidehty decreases (ah other things held constant), or if there is an obvious point of 
diminishing returns where further reductions in fidehty whl no longer have a significant 
effect. This wih again be useful in writing the AOIM controher module. It is expected 
that each time data fidehty is set to a lower value there will be a corresponding drop in 
bandwidth usage. 

The third test group is designed to compare the scalability of the AOIM to a 
comparable world with no AOIM. This test wih aid in determining the upper limit of the 
number of entities that can be in the world at once before bandwidth usage exceeds 
network capacity. 
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In every test, data was collected from one designated computer. The computer 
used for data collection had only one entity master created on it for each individual test. 
Every other entity on the test machine was an entity ghost representing an entity master 
on one of the population machines. 

AH other computers on the network were populated with multiple entity masters 
to simulate real world network interaction. This setup, with several machines, each 
having many entities, very closely approximates a real world scenario in which there are 
many machines, each with a single entity. This would correspond to a virtual world in 
which each user on the network controlled one entity on one computer. The main 
difference is that a single machine with many entities (40 entities for example) will use 
more CPU cycles for entity administration, and therefore will send data packets at a 
lower rate, than would 40 computers each with a single entity. 

In an ideal situation, these tests would have been mn on a network containing 
hundreds of machines, with only one entity master per machine. However, due to lim ited 
laboratory resources this compromise was used, and for showing the AOIM’s ability to 
reduce network traffic significantly, the compromise worked well. 

C. TEST CONFIGURATION 

The AOIM performance tests were conducted in the Naval Postgraduate School 
Virtual Reality Lab. All tests were conducted during periods of low network traffic to 
avoid large variations in testing which could have occurred due to uncontrolled traffic. 
The network in this lab consists of the following equipment: 

• 100 Mbps switched fast Ethernet LAN 

• Dell Dimension 4100 computers 

• 1 GHz Pentium El’s 

• 512 MB System memory 

• NVIDIA GeEorce II video cards with 64 MB of RAM 

• 3Com EtherLink XL 10/100 Ethernet cards 

• Windows 2000 Professional operating system 
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D. TEST GROUP ONE: INDIVIDUAL COMPONENT TEST 
I. Test description and parameters 

In the first test group, three different zone and population configurations, depicted 
in Table 1, were tested. 


Config 

Number of Entity Masters Loaded 

Interest 

Management 

Zones 

Test 

Machine 

Population Machines 

(total) 

1 

1 

15 

1 

2 

1 

15 

8 

3 

1 

30 

8 


Table I. Zone and population configurations 


For each of the three test configurations, several parameter settings were used 
(one change per mn) and a miming tally of BPS received by the test machine was 
recorded. After miming for one minute, the statistical package used contained an 
accurate one-minute miming tally of BPS received by the test machine. This tally was 
stored at ten second intervals and after thirty samples were gathered, the mean and 
standard deviation, for BPS received by the test machine, were logged. 

The parameters that changed for each new mn were: packet resolution and entity 
data fidehty. The number of entities in the world, and the zone configurations were held 
constant in each separate test. Table 2 depicts the parameter combinations used. 
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Setting 

Number 

Parameter 

Packet Resolution 

Data Fidelity 

1 

High 

High(l) 

2 

High 

Medium (20) 

3 

High 

Low (40) 

4 

Low 

High(l) 

5 

Low 

Medium (20) 

6 

Low 

Low (40) 


Table 2. Parameter combinations used during data collection 

As explained previously in Chapter IH, data fidelity can range from 1, which is 
the highest possible fidelity, to 40, the lowest possible fidelity. For these tests, however, 
only three settings were used: High (1), Medium (20), and Low (40). Again, fidehty 
affects packets per second, while resolution is float vs. double etc. 

2. Test results for test group one 

The results from test one, displayed in Table 3, show that of the two variable 
parameters, data fidehty has a much bigger impact on BPS than packet resolution does. 
On average, changing from high fidehty to medium fidehty resulted in a 74% reduction 
in mean BPS received at the test machine. Switching from medium fidehty to low 
fidehty resulted in a further 17% average reduction of mean BPS. 
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Packet 

Data 

Mean 

Standard 

Sample 

Resolution 

Fidelity 

BPS 

Deviation 

Size 

High 

High 

198,046 

3701 

30 

High 

Medium 

49,580 

862 

30 

High 

Low 

41,897 

2145 

30 

Low 

High 

182,839 

1098 

30 

Low 

Medium 

47,658 

711 

30 

Low 

Low 

38,904 

1049 

30 


Table 3. Test results for test configuration one 


Visually, there was practieally no difference in the behavior of the ghost entities 
viewed on the test machine as data fidehty dropped from high to medium. When fidehty 
dropped from medium to low, there was a shght degradation in smoothness of the ghost 
entities on the test machine. Despite this drop in quahty, which was noticeable, the 
system remained visually appealing. In fact, without having first seen the medium 
fidehty mode immediately before the low fidehty mode, it is doubtful a user would have 
known that significant dead reckoning was taking place. 

Next, with data fidehty held constant, packet resolution was varied with an 
average 6% reduction in BPS received at the test machine. There was no visual change 
in the performance of the ghost entities on the test machine. This result may or may not 
be significant. Because fish entities were used in ah tests conducted on this AOIM, and 
since fish do not maintain a constant speed, heading, or position, packet resolution 
probably is not as critical as it would be in other apphcations requiring more precision. 
In other words the results were visually plausible without necessarily being accurate. 

The results of the test using configuration two are depicted in Table 4. 
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Packet 

Data 

Mean 

Standard 

Sample 

Resolution 

Fidelity 

BPS 

Deviation 

Size 

High 

High 

22,114 

3,013 

30 

High 

Medium 

12,959 

7,081 

30 

High 

Low 

3,574 

978 

30 

Low 

High 

15,709 

4,124 

30 

Low 

Medium 

8,291 

5,814 

30 

Low 

Low 

4,777 

1,592 

30 


Table 4. Test results for test configuration two 


The results of this test, when compared to the previous test, were just as 
conclusive. Just simply breaking the world into eight zones of interest, and holding 
everything else constant, resulted in an 89% reduction in BPS received at the test 
computer versus no interest management at ah. Comparing ah hke parameter settings 
between configuration one and configuration two shows that breaking the world into 
eight interest management zones, results in an across the board reduction of BPS at the 
test computer of 86%. 

The data fidehty results of this test were different from the results of the 
configuration one test where most savings occurred by switching from high to medium 
fidehty. In the configuration two test, switching from high fidehty to medium resulted in 
a 44% reduction, and dropping from medium to low fidehty caused a further 57% drop in 
BPS received at the test computer. 

Changing packet resolution from high to low achieved an average BPS reduction 
of 10% at the test computer. 

The results of the test using configuration three (Table 1) are depicted in Table 5. 
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Packet 

Data 

Mean 

Standard 

Sample 

Resolution 

Fidelity 

BPS 

Deviation 

Size 

High 

High 

40469 

13991 

30 

High 

Medium 

6190 

2864 

30 

High 

Low 

4952 

2435 

30 

Low 

High 

24934 

8404 

30 

Low 

Medium 

4520 

2165 

30 

Low 

Low 

4514 

1156 

30 


Table 5. Test results for test configuration three 


The parameters for the last test in this series are the same as in test two exeept that 
the number of entities in the world has been doubled. Again, breaking the world into 
eight geographic zones p'oved effective. There was an 80% drop in BPS received at the 
test computer, as compared to the test with half the entities, but no interest management. 
When all test settings are compared between configuration one and configuration three, 
the result is an average reduction of 87% of mean BPS received at the test computer. In 
other words, despite the fact that there were twice as many entities in the world, the test 
computer saw an 87% reduction in BPS received. 

The data fidehty results were similar to those from test one. Switching from high 
to medium fidehty resulted in an 83% drop in mean BPS, while dropping from medium 
to low only caused another 10% drop. 

Packet resolution proved more effective in this test than it did in any other. 
Dropping from high to low resolution resulted in an average 25% drop in mean BPS. 
Again, this savings in bandwidth resulted in no noticeable degradation in entity visual 
appeal. 


37 







































E. TEST GROUP TWO: DATA FIDELITY PERFORMANCE TEST 

1. Test description and parameters 

In this test, which used only configuration one (Table 1), only the high packet 
resolution setting was used, and data fidehty varied from 1 to 40, one step at a time. At 
each setting, the statistical package was reset, and as before, BPS was recorded in ten- 
second intervals. Five samples per setting were collected. 

2. Test results for test group two 

The results of this test, shown in Figure 13, are interesting. As expected BPS was 
highest when data fidelity was set to 1. As data fidehty was decreased form 1 to 8, BPS 
dropped exponentiahy. As data fidelity was decreased further from 8 to 19, the decrease 
in BPS became mostly linear and much less pronounced. As data fidehty decreased from 
19 to 40, there was no appreciable drop in BPS received at the test computer. 
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F. TEST GROUP THREE: SCALABILITY TEST 

1. Test description and parameters 

The last test performed examined the BPS received in configuration one (Table 1) 
and setting one (Table 2), which is effectively no interest management at all. Interest 
management was then enabled, and more and more entities were added to the world 
(population machines only). The statistics package was reset every time the number of 
entities was increased, and average BPS was recorded at the test machine. The goal of 
this test was to find out how many entities could be added to the world as a whole, using 
aggressive area of interest management techniques, before the BPS received at the test 
machine exceeded that of a system with no AOIM and 16 total entities in the world. 

2. Test results for test group three 

Table 3 shows that with no interest management techniques apphed, and 16 
entities in the world, the test computer received an average of 198,046 BPS. These 
basehne values are what test three’s results will be compared to. 

The test was begun by inserting 85 new entities into the world, bringing the total 
number of entities to 101. All of the interest management tools previously described 
were enabled, and the stats package was given time to stabi liz e. The results were 
amazing! The mean BPS received at the test computer dropped to 19,385! With the 
number of entities in the world increased by over 600%, the AOIM was able to decrease 
network traffic at the test computer by over 90%. 

Though the initial goal was to drive the mean BPS back towards 200,000, and 
record the number of entities in the world required to do this, it quickly became apparent 
that it probably was not possible to insert that many entities into the world with the 
limited number of machines available. Given the results attained, this was a moot point 
anyway. These results show that the AOIM system can easily scale to allow a very large 
number of users. 

G. CONCLUSIONS 

The AOIM met or exceeded expectations in every test performed. The only 
surprising result was the fact that reducing data fidehty below 20 had no discernible 
effect. One explanation for this is that as data fidehty is set lower, the percentage of fuU 
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state packets being sent versus the number of simple steering commands being sent, goes 
down. Eventually there are so few, full state packets being sent, that they make up an 
insignificant number of the total packets being sent. Therefore, dropping the number 
even lower has no effect. It becomes analogous to removing a single white grain of sand 
from a large pile. There is definitely a point of diminishing returns. 

It should be noted that variance and standard deviation were much higher for 
some of the tests than for others. The reason for this is the type of entities used in the 
tests, namely fish, are all separate autonomous Agents, each with their own goals and 
characteristics. In addition, they die, and reproduce utihzing a genetic algorithm that 
produces new fish with characteristics of both its parents. Their goals can change at mn 
time, they can school (or not), and they can prey and be preyed upon. The end result of 
this is a large amount of geographic and temporal cohesion. In other words, a fish is 
likely to be in the same area it is currendy in, doing the same thing it is currendy doing, 
in the near future. So, for some of the tests, the fish may have been schooling in one 
area, and not sending many update packets over the network, and during other tests a 
shark may have been attacking the school causing the fish to scatter for their hves and in 
turn causing them to send a large amount of update packets. Some of the tests include 
data samples from both situadons resuldng in high variance. 

To test how the AOIM would perform in the real world, the decision was made to 
perform these tests with the enddes that actually populate the world, instead of wiidng 
new simple ones that just sent data packets at a constant rate. This decision is the reason 
for the high variance at times, but it is a much more reahsdc test of the AOIM’s 
capabihdes. 
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H. RECOMMENDATIONS 


As of now there is no mechanism in place for the AOIM to autonomously change 
the various settings tested here (see future work section in Chapter V). Based on the test 
results just described, however, the following is suggested as a mdimentary algorithm to 
implement autonomous behavior into the AOIM: 

• When BPS exceeds the upper threshold, decrease data fidehty one step at 
a time. 

• When data fidelity reaches 8, and the threshold is still exceeded, set packet 
resolution to low. 

• When the BPS threshold is exceeded once again, continue decreasing data 
fidehty one step at a time until it reaches 20. 

• If the threshold is once again exceeded, then simultaneously subdivide the 
world into eight new zones, reset packet resolution to high, and reset data 
fidehty to 1. 

• Continue performing these same steps in this order as necessary. If 
network traffic fahs back below the low threshold, simply reverse the 
steps taken, one step at a time. 

The goal of this algorithm is to maintain bandwidth usage at acceptable levels while 
at the same time maintaining visual appeal. Data fidehty and zone sphtting both offer 
large bandwidth savings, but data fidehty has a much smaher impact on visual appeal, so 
zone sphtting is used as a last resort. 

1. SUMMARY 

These performance tests have shown conclusively that the AOIM developed in this 
thesis is able to produce dramatic savings in network bandwidth usage in a peer-to-peer 
system, in real time, while at the same time maintaining system performance, and visual 
appeal. The two tools with the most dramatic savings potential are data fidehty, and the 
abihty to break the world into smaller zones of interest. Variable packet resolution, also 
offers significant, albeit less dramatic, savings. The real power in the AOIM comes from 
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being able to use these tools in combination to provide additional savings over using the 
tools separately. 
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V. CONCLUSIONS AND FUTURE WORK 


A. INTRODUCTION 

This thesis has demonstrated that it is possible to integrate an area of interest 
management scheme into a peer-to-peer virtual world supporting dynamic entity 
discovery. In addition, the support of dynamic class and protocol behavior control 
mechanisms provides greater scalability than a standard interest management scheme. 

Several innovations and techniques made this scalable interest management scheme 
successful. Using a combination of octrees and hash tables with their associated fast 
searches enables the system to grow quite large with no noticeable performance hit. An 
independent multi-layered approach allows the AOIM to constantly adapt to changing 
network conditions, by using each of its tools individually or in concert. Finally, the use 
of reflection allows the system to fuUy support dynamic entity discovery, a critical design 
goalofNPSNET-V. 

B. OCTREES 

Scaling interest management to a very large number of users required careful 
consideration of which data stmctures to use. There are two main reasons why the octree 
was the data structure of choice: speed and adaptability. The octree allows for fast 
changes to the configuration of the world. Zone collapses and zone expansions are easy 
to perform at mntime, with no noticeable drop in system performance or visual appeal. 
As for adaptability, octrees are a near perfect choice for representing three-dimensional 
space. 

Another benefit is the low memory requirement of the octree. Since the only 
information actually stored in each octree node is a String (zone name), and a Boolean 
(leaf node flag), the octree itself remains quite small, even as the world itself grows very 
large. This minimizes the delay a new player joining the world will experience while 
waiting for the octree to transfer over the network. 

Where the octree really shines, however, is when it performs a search. The search 

algorithm employed is effectively a three dimensional, binary tree search, with one query 

for each axis. Where a binary search can eliminate up to 50% of the possible solutions 
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with each query, this algorithm can eliminate up to 87.5% of the possible solutions per 
query (when the tree is fuU). This allows for extremely large worlds. 

As discussed in Chapter IH, the worst case search for octrees is 0(h), where h is the 
height of the tree. Assuming a circumference for the earth of 25,000 miles, and a 
diameter of about 8,000 miles, an octree height of about 18-19, would result in each leaf 
node being about 100 yards long per side; this assumes that the original octree cube 
enclosed the entire earth. At an octree height of about 25, this would reduce further to 
about a 1 yd cube. This means that an 0(h) search for even the largest environment is 
bounded by a reasonably sized h. 

Of course, just because the search is reasonable in theory, does not mean a world this 
large can actually be implemented. A completely full octree with a height of 25 would 
contain 8 nodes, and would subsequently require 8 multicast addresses. This is a taU 
order as 8^^ = 37,778,931,862,957,200,000,000. However, putting a sphere (the Earth) 
into a cube (initial octree) results in much wasted space, and the areas of the cube that do 
not contain any of the sphere could be culled. The result would be a vast reduction in the 
number of octree nodes actually required. 

C. MULTI-LAYERED APPROACH 

Another major contributor to this system’s success is the use of multiple 
independent layers. Chapter IV showed the effectiveness of the individual parts of the 
system, but the AOIM becomes even more powerful by being able to use the parts in 
combination with each other. 

For instance, using the Fish World example, if there were fifty fish in the world, 
and the mean bytes-per-second received at the test computer became too high, one way to 
deal with the problem would be to subdivide the world. However, if all fifty fish ended 
up in a zone together after the subdivision, then the net result would be that the bytes-per- 
second would stiU be too high. A standard, geographic based AOIM would stumble here, 
but the system developed in this thesis has additional weapons it can bring to bear. For 
example, the AOIM could respond further by reducing data fidelity, or packet resolution, 
while at the same time collapsing the eight new zones just created, since the subdivision 
tactic was not effective in this case. 
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The ability of this AOIM to respond to a highly dynamic environment, while stiU 
allowing tme peer-to-peer communication among the entities in the world, makes it 
unique. 

D. DYNAMIC ENTITY DISCOVERY AND REFLECTION 

One of NPSNET-V’s key features is that it supports dynamic entity discovery. 
This abihty to accommodate, previously unknown entities, seamlessly at runtime is a 
highly desirable feature that provides some interesting development challenges. 
Foremost among these is how to make fuU use of an entity’s capabihties without first 
knowing what those capabihties are. By utilizing the Java programming language and 
one of its capabilities known as reflection, these chahenges were overcome. Reflection 
aUows runtime queries of objects to determine their name, what methods they have, and 
what interfaces they implement. This feature allows the AOIM to query entities at 
mntime to ensure that they implement the NetworkTunable interface (necessary for 
the data fidehty feature to work properly). This lets the AOIM deal cleanly with entities 
whether they implement NetworkTunable or not. Without reflection, the AOIM 
would not have been able to fuUy support dynamic entity discovery. 

E. FUTURE WORK 

It has been said that good research asks more questions than it answers. This 
thesis is no exception. While this AOIM performs well, many improvements could be 
made to make it work even better. Some of the possible additions to this system that 
would greatly improve its performance are: a multi-agent monitoring system, a look¬ 
ahead algorithm, and support for multiple octrees. Lastly, due to NPSNET-V’s 
component architecture, an entirely new, non-geographic AOIM could be developed and 
inserted into the system. 

I. Multi-agent monitoring system 

As previously mentioned, there is currently no mechanism in place to 
autonomously change AOIM parameters at run time. AU parameters are currently 
manually controlled. Chapter IV suggested a rudimentary algorithm, based on testing, 
that should provide acceptable performance, and that could be quickly inserted into the 
AOIM. However, to fuUy reahze the potential of this system, a multi-agent control 
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mechanism could/should be developed, with the sole purpose of keeping the AOIM 
optimally configured for current network conditions. 

The AOIM developed in this thesis has the abihty to use all of its tools, at 
mntime, in any conceivable combination. These tools are effectively switches that can be 
flipped, and knobs that can be turned. No “if-then-else” algorithm could possibly hope to 
adapt to every possible network condition. The reason for this is simple; for an “if-then- 
else” algorithm to work, the person that writes it needs to allow for every possible 
network condition (at compile time), and that is just not possible. A multi-agent system 
however, with its abihty to learn and adapt to changing conditions, would be a prime 
candidate to flip the switches and twiddle the knobs. 

2. Look-ahead algorithm 

The AOIM, as currently implemented, only sends data to the zone that the 
sending entity currently occupies. This solution usuaUy works weU, but a problem 
develops when an entity is moving along a border between zones. As the entity enters 
the new zone, it begins sending and receiving packets accordingly, but if it quickly turns 
back into the old zone, and then back into the new zone once again etc, it ends up sending 
half of its packets to one zone and half to the other. This can cause anomahes such as 
pop-up. 

Pop-up occurs as an entity approaches a new geographic zone. Since the entity is 
not yet in that zone, it is not receiving any data packets from that zone, and in turn, there 
will be no other entities visible. The visible portion of the new zone will appear empty. 
However, as soon as the entity crosses the boundary and begins receiving new data 
packets, aU of the entities already in the zone will magically appear out of nowhere; they 
just “pop-up”. 

The fix for this problem is to use a look-ahead algorithm. If an entity gets within 
visual range of another geographic zone, simply subscribe to that multicast address as 
well. The entities will stiU pop-up in the world, but this may now occur beyond visual 
range of the entity in question, thus improving visual appeal. The drawback b this is the 
extra, potentially unnecessary, network traffic that will be generated. However, just like 
all the other tools the AOIM has, it can choose not to use them if necessary. If the 
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network is being crushed with traffic, the AOIM can decide to allow pop-ups to occur, 
thus saving bandwidth at the expense of visual appeal; this is just another switch the 
AOIM can fiip. 

3. Multiple octrees 

One additional feature that could be added to this system to improve fiinctionahty 
is support for multiple octrees. The benefit of multiple octrees is the ability to allow 
various levels of resolution simultaneously. For example, one octree might be broken 
down into many small zones of interest, which in the Fish World example would make 
sense if the water in the aquarium were very cloudy, and visibihty very low. Now 
suppose that an entity enters the world that does not rely on visibility at aU, but on 
something else such as sonar. This new entity can see across multiple zones (maybe aU 
of them at once) and, allowing a different octree to maintain its area of interest would be 
an elegant solution to the problem. The octrees themselves are so small, and the octree 
searches so fast, that many different octrees could be in use at once without causing a hit 
on system performance. 

4. Entirely new AOIM 

While a geographic AOIM is a good choice for many apphcations, there are 
situations where some other type of AOIM might be a better choice. NPSNET-V’s 
component architecture makes switching from one AOIM to another simple. This allows 
multiple AOIMs to be developed, each one completely different from the others, and the 
decision of which one to use to be made at mntime. Two possible types of AOIMs that 
could be developed are functional and temporal. 

A functional AOIM is one in which entity type, not location, determines what 
packets are received. An example of this would be a naval simulation in which ship 
entities receive packets from other ship entities, and on each ship, the individual sailors 
receive packets from other sailor entities on the same ship. However, ship entities never 
receive sailor entity packets. This type of system could work underneath the geographic 
AOIM to act as an additional packet filter. 

A temporal AOIM is a little harder to understand. An example of this type of 
system could be a world in which there are multiple time scales. The world itself could 
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be on a geological time scale for things such as seasonal changes, erosion, movement of 
continents etc. The entities in the world could be on a time scale in accordance with the 
human perception of how time passes. Lastly, there could be another time scale, call it 
‘Bullet Time’, where things happen in nanoseconds or picoseconds. An example of this 
is a bomb exploding. As far as the explosion is concerned, everything else in the world 
is moving so slowly as to really be considered stationary. 

F. CONCLUSION 

The main goal of this thesis was to show that it is possible to integrate an area of interest 
management scheme into a peer-to-peer virtual world that supports dynamic entity 
discovery. A further goal was to show that offering dynamic class and protocol behavior 
control mechanisms allows improved scalabihty over a standard interest management 
scheme. The AOIM developed in this thesis met these goals. 
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APPENDIX A 


A. INTRODUCTION 

It should be noted that the AOIM developed in this thesis is not a standalone pieee 
of software. It is tightly integrated into the NPSNET-V system. The eomplete NPSNET- 
V system, ineluding all souree eode and documentation, is freely available in a CVS 
repository located at: 

sse.cs .nps .navy .mil 

login = anoncvs 

password = PandaSD! (Please note the case) 

NPSNET-V is still very much a work in progress, and outside contributors to this 
project would be greatly welcomed. For more information about the system or for 
contact information, please point your web browser to 
http://www.npsnet. 0 rg/~npsnet/v/index.htn 1 l . 
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APPENDIX B 


A. INTRODUCTION 

This appendix provides UML diagrams representing the entire AOIM system. 
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Figure 14. UML diagram of the AOIM system (part I) 
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Figure 15. UML diagram of the AOIM system (part 2) 
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